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ABSTRACT 



An operator interface for a video editing system pro- 
vides a visual sense of the content of video sequences, as 
well as their length, while also providing enhanced 
interactive control of locations and time alignments of 
the video. As the video sequence is processed into the 
system, a smaD but representative sample of each frame 
is saved in a local memory, while the video itself is 
stored in mass storage. These samples are used to pro- 
vide a video pictorial timeline of die underlying stored 
video. The location of an operator's view into the video ' 
sequence is controlled by a cursor's movement along a 
detailed video pictorial timeline, a reverse motion area 
and a forward motion area to provide VTR control for 
location changes on the video tape. The cursor's move- 
ment can be controlled by a mouse or a knob. Icons, 
either static or dynamic, are produced within the mo- 
tion areas to indicate the amount of selected velocity. 
Timelines can be marked with time marks, roughly 
aligned and then automatically fine aligned by the sys- 
tem according to their respective time markers. The 
editing results associated with these timelines are also 
time aligned as a result of this process. 



10 Claims, 13 Drawing Sheets 
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quences as they arc being manipulated, but these simple 

. VIDEO EDITING SYSTEM OPERATOR timelines do not provide any visual cues as to the con 

INTER-FACE FOR VISUALIZATION AND tents of the video sequences. 
INTERACTIVE CONTROL OF VIDEO MATERIAL connccdon £th mther aspect 0 f the prior art, 

_ . . .. . „ 5 control of the displayed location on a video tape in a 

» » jontmaation of application Ser No. video tape recorder has traditionally been accomplished 

07/739.357 fled Aug. 2, 1991 and now abandoned. ^ ^ rotatablc whcch or ^ or a ^ OTe 

BACKGROUND OF THE INVENTION which operates in two modes, depending on the posi- 

This invention relates to video editing, and more 10 of a mode control switch^ In the first mode of oper- 
particularly to an operator interface for a video editing * tI0n * rota,50nal motlon of the Wfled or « lnto - 
system that provides enhanced visualization and inter- preted 85 P 0 ™ 01131 distance. In the second mode of 
active control of video sequences during the editing operation, the same rotational motion is interpreted as 
process. velocity magnitude control. U.S. Pat. No. 4,988,982 to 

Presently, the video editing equipment that is used 15 present inventor for "Touch Pad Machine Con- 
tends to interpose a lot of numbers and technical consid- trol", hereby incorporated by reference, describes this 
erations between an artist attempting to create a video prior art in its "Background of the Invention''. The 
program and the video images that must be manipny^ specification of the U.S. Pat No. 4,988,982 describes an 
in the course of creating that program. Cutting and alternative approach to V IK control that relies on a 
splicing film on a film editor provided a visual and 20 touch pad with different regions, one of which is de- 
spatial dimension to the temporal aspect of a fUm. Thus voted to position control, another of which is devoted 
far, efforts to bring this same sort of interactive conve- to velocity control, 
nience and "feel" to the video tape editing process have 

met with limited success, although considerable SUMMARY OF THE INVENTION 

progress has been made. 25 It is an object of the present invention to provide an 

U.S. Pat. No. 4,937,685 to Barker et al for a "Method operator interface for a video editing system that dis- 
of Display Presentation i for Video Editing;' discloses an p^y, a graphic representation of a video sequence to 
apparatus aiid inethod for vito composition that pres- give the operator a visual .sense of the content of the 
ents an operator with video labels at each end of each sequence, as we ]] ^ ^ length. 

video clip. Video labels are low resolution digital repre- 30 A I~S: . ~S^z7^ e .v • 

sentatioM of a frame of actual video. BecT^e these .According to one aspect of the invention, as the 
video labels contain much less total information than «d=° sequences first processed into the system, a small 
the underlying video frame that they represent, they bnt representative sample of each video frame is saved 
can be more readily stored and retrieved than actual m 8 10081 memor y. whfle ^ tt£elf » ^red » 
videoimages can be, and are therefore used as surro- 35 «^ The franu sarmjles are used m the display 

gates for the real video images during some parts of the to provide a video pictorial timeline of the underlying 
editing process. stored video. 

In the system described in the Barker eta] '685 patent, It is a further object of the present invention to pro- 
thc operator can view a clip of video or a transition vide an operator interface for a video editing system 
between two clips of video on a main monitor in an 40 that allows interactive and efficient positioning within 
"endless loop" repeating display. A set of four label the video sequence. 

display screens show the above described video labels According to another aspect of the invention, the 
of the cUps currently under active manipulation. A location of an operator's view into the video sequence is 
plurality of additional smaller display monitors are pro- controlled by a cursor in a machine control window, 
vided in vertical pairs for viewing a series of pairs of 45 The machine control window contains a detailed video 
video labels representing a series of clips currently ar- pictorial timeline, a reverse motion area and a forward 
ranged to be viewed in sequence. motion area. The interaction of the cursor with the 

In the Barker et al '685 system, the operator is able to reverse motion area produces selected amounts of re- 
manipukte the video labels instead of the v«ieo itself, verse v(tlocIty ai 0Ilg the video sequence seen in the 
thereby permitting the re^gement of the video 50 vtfeo pictorial timeline and on otheTparts of the screen. 

^T^ri'rrJM thC - aCt0al ^Lf 1 Similarly, the interaction of the cursoVwkh the forward 

stored m its original order, thus mmirmzmg the need for ^ , . , , _ 

last access to ttevideo itself. WhTrapMh access to the ^^«ca produces selected amounts of forward ve- 

actual video is required, it is achieved by recording ^TlL S ^ w ^ ch <*» 
multiple copies of the same video clip on multiple video 55 te ^ dynamic, are produced within the mohon 
tape recorders {VTRs) and posWoning each of the 81683 to mdlcate amount of se,ected veloaty. The 
VTRs at different locations on the clip, so that one cxmoT 08,1 """rolled by either a mouse or a knob. 
VTR is always close to any desired point of interest. 11 K 8 ftmher ob j ccl of present invention to pro- 
While the Barker et al '685 system provides the oper- ^ 8 meth od 01 video to Pe recorder position and vc- 
ator with video labels at the beginning and end of clips, 60 \ ocis ?. contro1 ^ makes such control easier and more 
a sequential continuous loop display of the active clip intuitive for the operator. 

and a readout of the duration of the active clip, it does According to another aspect of the invention, a single 

not provide any graphic display of the clip's length or knob has three regions, one for tape position control 

the rest of its internal contents. and one each for forward and reverse tape velocity 

A video editing software system called "Media 65 control. Rotational motion of the knob into the latter 

Maker" is produced by Macromind Inc., 410 Townsend two regions produces velocity changes, with the cur- 

St, Suite 408, San Francisco, Calif. This system displays rent velocity being determined according to the follow- 

simple timelines that indicate the length of video se- ing relationship: 
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where, 3 

Vsris the current velocity, 

Vatax is the maximum velocity, 

Dear is the current distance into the region, and 

Dam is the maximum distance into the region. 

It is another object of the present invention to pro- 
vide an operator interface for a video editing system 
that automatically performs' fine tim* alignment of two 
video sequences that have been marked and coarsely 
aligned by an operator. 

According to another aspect of the invention, two or 1J 
more timelines can be marked with time marks and the 
video editing system then automatically performs a fine 
iimr alignment of the timelines according to their re- 
spective time markers. When an editing session is com- 
pleted, the results of that editing session reflect the time 20 
synchronization produced by this time alignment pro- 
cess. 

The subject matter of the present invention is particu- 
larly pointed out and distinctly claimed in the conclud- 
ing portion of this specification. However, both the 25 
organization and method of operation, together with 
further advantages and objects thereof, may best be 
understood by reference to the following detailed de- 
scription and accompanying drawings. 

BRIEF DESCRIPTION OF THE DRAWINGS 30 

FIG. 1 is a screen view of a portion of the operator 
interface of the present invention; 

FIG. 2 is a simplified block diagram of the video 
editing system; 35 

FIG. 3 is a block diagram representation of how a 
video pictorial timplinp is created; 

FIG. 4 is a block diagram representation of ho w time 
marks and cursors are generated; 

FIG. 5 is a block diagram representation of how a 40 
machine control window operates; 

FIG. 6 illustrates an implementation of a machine 
control window utilizing a potentiometer instead of a 
mouse as the means for operator input; 

FIGS. 7A and 7B illustrate the process of marking 45 
and time-aligning two video sequences to achieve time 
alignment between them; 

FIG. 8 is an illustration of three simple timelines and 
two video pictorial tjTnflin^g in a single Workspace; and 

FIGS. 9 A, 9B, 9C, 9D and 9E are a simplified logic SO 
flow diagram of a portion of the operator interface for 
a video editing system in accordance with the present 
invention. 



DETAILED DESCRIPTION 
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The display shown in FIG. 1 is part of a user interface 
implemented on ah Apple Macintosh computer. It uti- 
lizes window and cursor control approaches that are 
. somewhat similar to the normal Macintosh (TM) inter-, 
face, but instead of the normal Macintosh keyboard this 60 
system employs a keyboard that has colored keys. That 
keyboard resembles the keyboards and color codings 
that are somewhat standard in the television post-pro- 
duction editing equipment industry. 

The screen shown in FIG. 1 is divided into four quad- 65 
rants. The upper left quadrant contains most of the 
features associated with the present invention and will 
be discussed in detail below. The upper right quadrant 



is a Workspace 15 that has a vertical layers axis 25 along 
its left side and a horizontal time axis 24 along its top. 
The scale on horizontal time axis is variable, allowing 
the operator to select the time span that will be dis- 
played. 

The Workspace 15 shows, for a portion of a program 
being created, the sequential arrangement of video seg- 
ments or clips, as well as the relative synchronization of 
those video crips (layers) that are to play at the same 
time. The Workspace 15 is divided into two parts, a 
background area 33 on the bottom and a layers area 32 
on top. The background area 33 is used for basic editing 
that only involves a linear arrangement of video sequen- 
ces with cuts or dissolves. The material that appears 
here may form the background for a more complex 
event that is organized into layers in the layers area 32. 
In the background area 33, video segments are shown in 
.two ways: they either appear as a wide tfmrfiiw-, such as 
the one whose dark portion is labeled "Blk" and whose 
shaded portion is labeled "B Vtr", or as a title, such as 
the darker shaded timeline above the "B Vtr" timeline, 
the one with the sloping leading edge that represents a 
transition. Audio sources associated with the wide time- 
line appear as narrow rimplmw beneath it 

In the layers area 32, video segments are shown as 
simple, narrow timelines 21. Transitions or effects in the 
layers area 32 are shown as separate individual time- 
lines. Layers in the Workspace 15 that are currently 
included in the edit are shown shaded, as layers L3 and 
L2 are, while layers that are not currently included in 
the edit are shown in white (unshaded), as layers L5 and 
L4 are. 

The lower right quadrant of the screen shown in 
FIG. 1 contains a Text Timeline 35 that provides a 
textual display of the composite timeline being devel- 
oped in the Workspace 15 areas 32 and 33. The lower 
left quadrant of the screen displays an Edit (Decision) 
List 36 (EDL) of editing previously performed. The 
Edit List 36 area can be scrolled to look at old versions 
of edits or at a partial result of the current editing pro- 
cess being performed in the Workspace 15 areas 32^53. 
As work is completed in the Workspace 15 areas 32,33, 
it is appended to the current Edit List 36 at the location 
of vertical reference line 29. 

As can be seen in FIG. 1, the first ten seconds of an 
edit is shown in the Workspace 15 layers area 32, the 
background area 33 and the Text Timeline 35. At the 
left side of the Workspace 15, five seconds of pre-roO 
time for the switcher is shown. Following that, in the 
background area 33 and in the Text Timeline 35, two 
seconds of black background is shown. Audio 1 (Al) 
and Audio 2 (A2) are also black during this time, indi- 
cating a silence on the audio tracks associated with the 
black video. The interval of black video and silent audio 
is shown in the Text Timeline 35 as the second entry, "0 
CUT BLK". The first entry in the Text Timeline 35 
indicates that this recording (RECD) will use Audio 1, 
Audio 2 and Video (A12V), and that it will be appended 
to the Edit List 36 at time 10:00:00:00 in the Edit List 36 
frame of reference. • 

The third entry in the Text Timeline 35 indicates that 
at time two seconds, 2:00, there will be a dissolve, 
DISS, lasting sixty frames, 60F, to the B VTR which is 
to start at time 2.-00:12.-03 in the B VTR frame of refer- 
ence. This same information is shown in the background 
area 33 by the white box with the crossed lines as diago- 
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nals. Here the duration of the dissolve is shown with the 
notation "60/'. 

The fourth entry in the Text Timeline 35 indicates 
that at time four seconds 4:00, another source, E VTR, 
will PLAY starting at time 133$S33 in die E VTR 5 
frame of reference. This same information is shown 
visually in the layers area 32 on layer L2. 

The fifth entry in the Text Timeline 35 indicates that 
at time 4:00 the K'SCOPE (a Kaleidoscope special 
effects device) is to play. The same event is shown in 10 
the layers area 32 on the layer below layer LI, the layer 
labeled "KSc". 

The sixth entry in the Text Timeline 35 indicates that 
at time 5:15 a TTTLE-A will be KEYed in over an 
interval of 10 frames, 10F. This same information is 13 
presented in the background area 33 by the dark bar 
with the diagonal leading edge that is located above the 
timcrinft labeled "B Vtr". The diagonal edge indicates 
that the title dissolves in, with the length of the dissolve 
being represented by the length of the diagonal portion 20 
of the bar. At the same time as the title is keyed in, 5:15, . 
a new audio source, A3, is shown entering the record- 
ing by the entry REGD A123V. At time 6410 the C 
VTR is to start playing, and at time 7:20 its MOTION 
is changed to 50% of normal playing speed. At time 23 
7:10 audio source A3 is seen to end with the entry 
RECD A1ZV. 

Referring now to the left side of FIG. L two video 
pictorial timelines 10 and 11 according to the present 
invention are visible. A main video pictorial timeline 10 30 
appears in the region that separates the upper left quad- 
rant and the lower left quadrant A more detailed video 
pictorial timeline 11 appears above the main video pic- 
torial timeline, in the lower portion of the upper left 
quadrant These video pictorial timelines provide the 35 
operator with a sense of the image presented by the 
video sequence. As will be further explained below, 
when a new video sequence Is put into the video editing 
system of the present invention, a sample is taken of 
each frame. This sample is small, typically only eight 40 
pixels of information, but when these frame samples are 
assembled into a video pictorial timeline 10,11, this 
limited data provides enough information to visually 
cue the operator quite effectively. 

The main video pictorial timeline 10 shown in FIG. 1 43 
represents a video sequence that can vary in length 
from a few seconds to an hour and a half. The desired 
scaling factor is produced by duplicating or decimating 
the frame samples representing the video sequence so 
that a representative sample of the samples is displayed SO 
in the video pictorial timeline 10. Immediately above 
the main video pictorial timeline 10 there is an associ- 
ated main audio level timeline 22 As the name suggests, 
the audio level timeline 22 illustrates the audio ampli- 
tude along the main video pictorial timeline 10 by dis- 33 
playing a single-sided "envelope" indicative of the am- 
plitude level at each point in time. 

In either of the video pictorial timelines 10,11 or the 
simple timelines 21 in the Workspace 15 areas 32,33, 
clips can be taken from a longer video sequence by 60 
clicking a control button on the mouse or other cursor 
control device while the cursor 12 is on the mark In box 
26 to identify a starting point or the mark Out box 28 to 
identify an ending point Reference marks 31 for per- 
forming time alignment between different pieces of 63 
video can be placed anywhere within a video sequence 
using the Ref Mk box 30 or a corresponding key on the 
keyboard (66 in FIG. 2). 
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Video window 14 displays the video image at the 
current location within any video sequence that is acti- 
vated, including the composite video segments that are 
the result of an editing session. The video window 14 
displays the scene at the location of cursor 12 along the 
video pictorial timrlinp 10, whether it is still or moving, 
at normal speed or higher speeds. Immediately to the 
right of the video window 14 is a source status display 
34. Six VTRs, A through F, are listed as sources, with 
squares shown to the left of those that are stopped. 
Sources that are playing forward at normal speed, such 
as the B source, are indicated with a **>" pointing to 
the right A ">>" indication signifies fast forward, 
while a " < < " indication to the left, as is shown for 
source A, rndiratps fast rewind. Below the list of 
sources there is a RECord box 37, a PreVieW box 38 
and a palette of most used transitions 39. niriring the 
control button on the mouse while the cursor 12 is 
positioned on the RECord box 37 causes the activity 
represented by all of the active layers in the layers area 

32 and the background area 33 to be recorded into the 
recording medium. Clicking on the PrcVicW box 38 
causes ■a "virtual edit" of the active layers to be pre- 
viewed as a composite sequence but not recorded, Le., 
be made visible in the video window 14 as they would 
appear if recorded. Clicking on cither the "30" (frames) 
box or the "60" (frames) box in the palette of most used 
transitions 39 allows the user to drag a transition of that 
length to the location of his or her choice in the Work- 
space 15 areas. 

The content of the main video pictorial timeline 10 
changes to reflect whichever video sequence has most 
recently been idrntififd as the current, active one by the 
operator clicking the control button on the mouse 70. 
The current active video sequence can be designated 
either by clicking on its source in the status area 34 or 
on its simple timeline 21 in the Workspace 15 areas 32 or 

33 in the upper-right quadrant or by the use of keys on 
the keyboard. The main video pictorial timeline 10 can 
display active video sequences that are a single clip of 
video or, when the preview (PVW) bos 38 is clicked 
on, the main video pictorial timeline 10 can display a 
composite sequence of video that has been virtually 
edited into a single sequence in one of the work areas 32 
or 33, as was explained above. 

The user's movable cursor 12 can be locked onto one 
of the cursors (also 12 in this Figure) that are perma- 
nently associated with the main video pictorial timeline 
10 and the detailed video pictorial timeline 11 by mov- 
ing the movable cursor over the associated cursor and 
eliciting the control button once. The user's movable 
cursor can be unlocked from the cursor 12 associated 
with one of the video pictorial timelines 10,11 by click- 
ing the control button twice. 

While the user's movable cursor 12 is locked to one of 
the cursors associated with a video pictorial timeline 10 
or 11, moving the cursor 12 causes one of two results, 
depending on whether the mass storage means {hem 50 
in FIG. 2) is linear, such as a conventional video tape 
recorder, or random access, such as laser discs or multi- 
ple hard discs. As will be further described with refer- 
ence to FIG. 5, if the system's mass storage means 50 is 
linear, moving the cursor 12 faster than the linear stor- 
age means can move the video tape causes a second 
cursor (13 in FIG. 5) to appear. The second cursor 13 
shows the actual VTR position on the tape, while the 
cursor 12 under the operator's control represents the 
desired tape position. When the cursor 12 slows down 
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or stops, the other cursor, representing the actual poai- ward motion area 20 to the right of the detailed video 
tion. moves to catch op with the desired position at the pictorial tm mfi nft 11, the image in the video window 14 
maximum speed of the linear storage means 50. If the fast forwards and the scenes represented by the frame 
system's mass storage means SO is random access mem- samples in the detailed video pictorial timeline 11 
ory of some kind, no second cursor appears when the 5 stream to the left 

cursor is moved rapidly and the video window 14 re- The speed of these forward and reverse motions de- 
flects the cursor 12 position almost instantaneously. pends on how far beyond the ends of the detailed video 

If the user dicks the mouse 70 button while the cur- pictorial timeline 11 the movement the desired 
sor 12 is on the nuun picton&l tinralm^ 10, but not on the position cursor 12. When the desired position cursor 12 
cursor 12 arav -ia r r rf with the main pictorial timeline 10, 10 reaches the outside end of either of the motion areas 
the cursor 12 associated with the main pictorial timeline 18,20, (he physical tape speed is maximum in that direc* 
10 becomes a current position cursor 13 and the cursor tion. As the magnitude of this attempted movement 
12 at the position determined by the mouse 70 becomes increases, an indicator (to be further described in con- 
a desired position cursor 12. If the mass storage means nection with FIG. 5) fills the forward motion area 20 or 
50 is random access, the current position cursor will IS the reverse motion area 18 in proportion to how fast the 
almost instantaneously move to the desired position and video sequence is moving relative to its maximum' 
the image at that location will be displayed in the video speed. When this indicator fills the forward or reverse 
window 14. If the mass storage means 50 is linear, e.g., motion areas 20 or 18, the velocity of the video se- 
VTRs, the current position cursor 13 will move toward quence movement is at its maTirmim value, 
the position of the desired position cursor 12 at the 20 When the operator wants to slow down or stop the 
maximum speed of the VTR and the images along the motion of the video sequence through the detailed 
way will appear in the video window 14. video pictorial timeline 11, he uses the mouse to move 

The more detailed video pictorial timeline 11 directly the desired position cursor 12 back towards the detailed 
above the main video pictorial timeline 10 is part of a video pictorial timeline 11 portion of the machine con- 
"machine control window". The other parts of the 25 trol window display, thereby slowing the velocity of 
machine control window are the reverse motion area the video sequence motion. Completely returning the 
18, the forward motion area 20 and the desired position desired position cursor 12 to the detailed video pictorial 
cursor 12. The detailed video pictorial timeline II con- timeline 11 stops the motion of vkleo sequence entirely, 
tains a four second interval of video, with each Video Referring now to FIGS. 2 and 3, the frame samples 17 
frame sample shown two pixels wide for clarity. A 30 that make up video pictorial Hn H tn rs 10 and 11 are 
machine control window audio level timeline 23 shows produced by a digital video and audio processor 58 and 
the amplitude of the audio associated with the video stored in a local memory 60 that is convenient to the 
shown in the detailed video pictorial timeline IX A time graphic interface display generator 57. As will be fur- 
mark 31 seen above the m ac hine control window audio ther described below, time marks 31 and single video 
level timeline 23 marks a particular location in the de- 35 frames 19 are also stored in the local memory 60. 
tailed video pictorial timeline 11 and its associated ma- The underlying video frames 19 that the frame sam- 
chine control window audio level timeline 23. pies 17 represent are stored remotely in mass storage 

The machine control window permits the operator to means 50. The video frame samples 17 are stored with 
scroll slowly through individual frames of the currently pointers to the locations of the corresponding video 
active video sequence, whether the storage is linear or 40 frames 19. These pointers, depending on the nature of 
random access (non-linear). It is used for fine control to the mass storage means 50, are «tW the SMPTE time 
identify and mark individual frames in the video se- codes, or memory addresses or both. While the icons 
quence with time marks 31 for use in time aligning shown for the mass storage means 50 in FIG. 2 suggest 
different video sequences. As the desired position cur- VTRs, they in fact can be any other video storage 
sor 12 is moved along the detailed video pictorial time- 45 means and are preferably high capacity random access 
line 11, its motion is reflected in the content of the video memory, such as multiple laser discs or magnetic hard 
window 14 and in the position of the cursor 12 associ- discs. 

ated with the main video pictorial timeline 10, although The artist or editor operator controls operation 
it should be noted that the motion of the latter may be through a keyboard 66 and a mouse or other cursor 
nearly imperceptible depending on the scale factor of 50 control device 70 while interacting with the CRT dis- 
the main video pictorial timeline 10 and how that relates play 68. Keyboard commands are processed by the 
to the view provided by the detailed video pictorial human interface logic portion of block 64 which also 
timeline 11 in terms of the ratio of their time scales. contains manual controls. As will be further described 
To access a location that is beyond one of the ends of below, most of the activity associated with the present 
the detailed video pictorial timeline 11, the desired posi- 55 invention occurs within the graphic interface display 
tion cursor 12 is moved past that end. Moving beyond generator 57, the digital video and audio processor 58 
the ends of the detailed video pictorial timeline 11 and window and screen display generator 62. These 
causes the detailed video pictorial timel i n e 11 and its interact with local memory 60, cursor control device 70 
associated audio level timeline 23 to start moving in the and the CRT display 68 to effect most of the behaviors 
opposite direction to bring the desired location into 60 associated with the invention. 

v * ew - Operator input from the keyboard interacts with 

If the movement of the desired position cursor 12 is manual control human interface logic 64 to produce 
into the reverse motion area 18 to the left of the video desired effects in the editor logic and automatic control 
pictorial ti mri inc IX the video image in the video win- software 54. The video source interface 56 interacts 
dow goes into reverse motion as the scenes represented 65 with the editor logic and automatic control software 54 
by the frame samples in the detailed video pictorial and the manual control human interface logic 64, re- 
timeline 11 stream to the right Conversely, if the move- ceiving instructions from them and returning status 
ment of the desired position cursor 12 is into the for- information. Responsively to these instructions, the 
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video source interface 56 controls access to the contents 
of the mass storage means 50 via selecting means for 
video sources 52. The selecting means for video sources 
52 stores (and retrieves) timecode and control data and 
video and audio information in (and from) the "«« 5 
storage means 50. The video source interface provides 
identification of the video source, the current video 
frame data and the timecode. range to be displayed to 
the graphic interface display generator 57. The graphic 
interface display generator 57 controls the window and 10 
screen display generator 62 to produce the display on 
the CRT display 68. 

Referring now to FIG. 3, the digital video and audio 
processor 58 produces a frame sample 17 for each video 
frame 19, and sends these frame samples to the local is 
memory 60. In one embodiment, each frame sample 17 
that is taken consists of eight individual pixels distrib- 
uted vertically along the center of the video frame 19. 
In another embodiment, the frame sample 17 consists of 
eight composite pixels that each represent the average 20 
. video values within eight different regions distributed 
vertically along the center of the video frame 19. In 
either case, these frame samples 17, when displayed in 
sequence as a video pictorial timeline, represent an 
"edge view" of the contents of the underlying video 25 
sequence. 

Other methods of producing the frame samples 17 are 
envisioned. One of these requires considerably more 
processing, but provides the operator with an even 
more intuitive feel for the activity occurring within the 30 
video sequence being represented. This approach relies 
on an algorithm to identify the most dramatic features 
of the luminance and chrominance values found in a 
frame 19 and emphasize those in the edge view repre- 
sentation. Even when these frame samples 17 are the 35 
result of sophisticated processing and communicate 
more content to the viewer, they are very small in terms 
of the data required compared to the data content of the 
video frame 19 that they represent 

Audio recordings that are input to the system, 40 
whether or not they are directly associated with a video 
recording, are processed to produce audio envelopes 
22,23 that provide a visual representation of the audio 
activity over time. The data required to reconstruct 
these audio envelopes is also stored in the local memory 45 
60. Audio is sampled 60 times per second and stored as 
eight bits of data. The audio envelope data is also stored 
with pointers to where the underlying audio is stored in 
the mass storage means 50 (FIG. 2). 

The video source interface 58 contains a pictorial 50 
timeline generator 59 and a current tape/source position 
detector 55. The video source interface 58 receives 
source identification and zoom control (time range to be 
displayed) input from the human interface logic 64 and 
timecode, video and audio information from the mass 55 
storage means 50 (both shown in FIG. 2). Based on the 
zoom control information, the pictorial timeline genera- 
tor 59 determines how much the video frame sample 
record must be decimated for display. Changing the 
zoom control alters the dynamic range of the video 60 
pictorial timeline 10 from ninety minutes to a few sec- 
onds. It also correspondingly changes the detail that can 
be seen as a result of the video frame sample decimation 
that must occur to provide room for the longer times. 

The current tape/source position detector supplies 65 
the graphic interface generator 57 and the pictorial 
timeline generator 59 with information about the cur- 
rent position on the video tape or other source. The 
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pictorial timeline generator 59 controls which video 
pictorial timeline 10,11 and audio level timeline 22^3 
data the local memory 60 is to supply to the graphic 
interface generator 57. Using the current tape/source 
position information provided by the current tape/- 
source position detector 55 and the pictorial timeline 
data supplied from the local memory, the graphic inter- 
face generator 57 operates through the window and 
screen display generator 62 to create the video pictorial 
timrlinrs 10 and 11 and the audio level tjra^jp wi 22 and 
23 on the CRT display 68 (FIG. 2). 

FIG. 4 is a block diagram representation showing 
how cursors 12,13 and time marks 31 are produced and 
applied to the video pictorial timelines 10 or 11. As 
partially described above, one cursor 12 is controlled by 
the operator to i ndica t e the desired position on the 
video sequence, while a second cursor 13 indicates the 
actual position of a video tape recorder or other linear 
mass storage means 50 oa the video tape or other me- 
dium. 

The operator controls cursor 12 via the mouse 70 or 
other cursor control device and the user's cursor gener- 
ator 72. A cursor position detector 74 monitors the 
cursor 12 position and provides timeline position com- 
parator 76 with the user's cursor 12 position. The time- 
line position comparator 76 also receives current posi- 
tion on the rimriinii information from the timeline cur- 
rent position location detector 61. By comparing these 
two inputs, the timeline position comparator 76 is able 
to generate position error information. A timecode 
translator 77 converts the position error information 
from the timeline position comparator 76, which is in 
the frame of reference of the CRT display (68, FIG. 2), 
into desired location changes in timecodes which are in 
the frame of reference of the stored video. These de- 
sired changes are then provided to the video source 
interface 56 which sends a corresponding motion com- 
mand to the VTR or other source. 

The video source interface 56 receives timecode and 
status information from the VTR or other source. This 
current source position information is detected by the 
current tape/source position detector 55 within the 
video source interface 56 which supplies it to the time- 
line current location detector 61 and to the graphic 
interface generator 57. The timeline current location 
detector 61 supplies current position ou the timeline 
information to the timeline position comparator 76, as 
mentioned above, and to the cursor generator for 
source position 80. The cursor generator for source 
position 80 produces the current position cursor 13 on 
the CRT display 68 at the location indicated by the 
current position on timeline information. 

In addition to the current source position information 
menti oned above, the graphic interface generator 57 
also receives time display range (zoom) information and 
"Mark" command information from user input When a 
mark command is received, the graphic interface gener- 
ator 57 produces a time mark 31 at that location. This 
time mark 31 is stored in the local memory 60 along 
with the frame samples 17 that constitute the video 
pictorial tirnHTnre 10,11 so that future video pictorial 
timelines will include this time mark 31 at this location. 
The graphic interface generator will continue to gener- 
ate a time mark 31 at the stored location until it receives 
a delete command (not shown) while the cursor posi- 
tion detector 74 is indicating the position of this old 
mark. 
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FIG. 5 is a block diagram representation that shows knob 81 as active knob position indicator 79 moves 
how the machine control window operates. A desired . through a positional region 85. 
position cursor 12 is controlled using the moose or other Sufficient counter-clockwise rotation of the control 
cursor control device (70 in FIG. 4) and the user's cur- knob 81 takes the active knob position indicator 79 out 
sor generator (72 in FIG. 4). This cursor position is S of the positional region 85 and into a reverse region 83, 
monitored by the cursor position detector 74. The out- while sufficient clockwise rotation of the active knob 
put of the cursor position detector 74 is monitored by a position indicator 79 takes the active knob position 
variable rate detector 86. Based on whether the desired indicator 79 out of the positional region 85 and into a 
position cursor 12 is in the reverse motion area 18 or the forward region 87. Maximum counter-clockwise rota- 
forward motion area 20 and how far into one of those 10 tion of the control knob 81 causes the desired position 
areas the desired position cursor is, the variable rate cursor 12 to move to the left end of the reverse motion 
detector produces forward and reverse velocity control area 18 and produces, as in the other embodiment, maxi- 
signais that inform the video source interface 56 and the mum reverse velocity of the VTR or other video stor- 
variable rate icon generator 88 as to the user's desired age device. Similarly, clockwise rotation of 
velocity. The video source interlace 56 in turn produces " the control knob 81 causes the desired position cursor 
forward and reverse rate commands to control the ao . 12 to move to the right end of the forward motion area 
tual mass storage mwlirrm tape or other source. 20 and produces maximum forward velocity along the 

The variable rate, icon generator 88 generates either a video sequence, 

reverse variable rate icon 89 or a forward variable rate The velocity of the video tape mar-imB is propor- 

kxra 90, depending on the direction and rate of desired 20 tional to the rotational distance of the active knob posi- 

motion. In the example shown in FIG. 5, the reverse tion indi ca t or 79 into the forward or reverse regions 87 - 

variable rate icon 89 has grown larger toward the left to or 83, with maximum rotation corresponding to the 

fill part of the reverse motion area 18 in proportion to maximum forward or reverse velocity of the video tape 

the amount of the rnmrimnm available reverse velocity machine: 
that is being used. In this example, the size of the reverse 25 
variable rate icon indicates that about 40% of the mad- 

mum available reverse velocity is being used. When Vaa ~ "ZwT " Vma * 

forward motion is involved, the forward variable rate 

icon 90 operates in the same way, extending to the right ^ where, 

to fill the forward motion area 20 in proportion to Va, is the current velocity, 

amount of the maximum available forward velocity that Vnua is the maximum velocity, 

is being used. D rar is die current distance into the region, and 

As the current location along the active video se- &mcx 13 the mflxiTppm rfwfunoff into the region, 

quence changes, the detailed video pictorial timeline 11 3J The software of the human interface system can 

produced by the graphic interface generator 57 moves know the rotational position of the active knob position 

beneath the current position curser 13 to keep it posi- indicator 79 by any of several means. If the knob 81 

noned over the current location. To accomplish this, employs a wire wound potentiometer, the output volt- 

the graphic Interface generator 57 needs information age indicating its position can be translated into a digital 

indicative of the current position and the tintecode 40 number by an analog-to-digital converter. Alteraa- 

range, so that it can ascertain which frame samples (17 lively, the knob 81 can include an optical position en- 

in FIG. 3) it must retrieve from the local memory $0 in coder to provide a direct digital readout of its position, 

order to produce the correct detailed video pictorial The logic required to interpret the digital knob position 

ti m e line 11. It receives the current tape/source position information is similar to that described for cursor posi- 

information from the current tape/source position de- 4.3 tkm detection in connection with FIGS. 4 and 5. 

. tector 55 within the video source interface 56". Ideally, the locations on the knob 81 between the 

While the foregoing discussion concerning the opera- positional region 85 and the forward and reverse ra- 
tion of the desired position cursor 12 in the wifiwtw gions 83 and 87 are provided with detents to give the 
control window has referred to the motion of the cursor operator a slight amount of tactile feedback as the tran- 

12 into the forward and reverse motion areas 20 and 18, so shion is made from positional operation to velocity 
that motion need not actually occur. In one implemen- controlling operation. The knob 81 also has stops at the 
tation, when the mouse 70 is moved in a way that would outside ends of the reverse region 83 and forward re- 
cause the cursor 12 position to go beyond the edge of gion 87. 

detailed video pictorial timeline 11, the apparent posi- Another difference between the embodiment of the 
tion of the cursor 12 stops at that edge and yet the 55 machine control window shown in FIG. 6 and the one 
virtual motion of the cursor 12 can still be detected from shown in FIG. 5 is unrelated to the difference between 
additional real motion of the mouse 70. using a control knob 81 rather than a mouse 70, and 
FIG. 6 shows an alternative embodiment of the ma- could be used with either embodiment The reverse 
chine control window input mechanism. In this embodi- variable rate icon 89' and forward variable rate icon 9C 
ment, the desired position cursor 12 is controlled by 60 in this implementation are both static, whereas the ones 
control knob 81 instead of a mouse or other cursor described above were dynamic. Whereas the icons de- 
control device 70. As before, when the desired position scribed above changed size, extending dynamically to 
cursor 12 is moved back and forth along the detailed indicate various velocities, the ones shown here do not 
video pictorial timeline 11, the current position cursor change, but nonetheless still suggest the idea of increas- 

13 follows and the frame selected is displayed in the 65 ing velocity by their sloped shape and the motion of the 
video window (14 in FIG. 1). This motion of the desired desired position cursor 12 along that shape, 
position cursor 12 within the detailed video pictorial One more difference between FIG. 6 and FIG. 5 is 
timeline 11 is controlled by the motion of the control that in the implementation shown in FIG. 6 the current 
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position cursor 13 is disposed above the detailed video processed 104 and it is determined whether a new 

pictorial timeline 11, rafter than below it, as it was in source has been selected 105. 

FIGS. 1, 4 and 5. In this implementation the current If a new source has been selected, 105- Yes, the new 
position cursor 13 remains visible even when it catches video is displayed 106 in the video window 1*. A new 
op with the desired position cursor 12, whereas in the 5 main video pictorial timeline 10 and detailed video pic- 
other implementation, the two merge when they are tonal timeline 11 are created 107 and 108, as described 
" "g 1 ? 1 - . _ in connection with FIG. 3. 

Refemng now to FIGS. 7A and 7B, we see two Following these steps, as shown in FIG. 9B, a deter- 

snnple timelines 21-A and 21-B extending horizontally ruination is made 109 as to whether or not the videotape 

along a tone axis 24 and displaced vertrcaHy along a 10 machine or other video source is in motion. If no new 

layers axis 25 The operator deans to tone align a point ^uict ^ selected, 105-No, this determination 

of hmehne 21-A with a pom on tundme 21-B. Uang a 109 is made directly afterthe determination of step 105 

^ £T ^^.f ^J^T * *e machined motion. 109-Yes, thc^SpS 
of what has now become the achve toe mark 27 the tion - a ^ Hw l nTc^ection 

operator has previous^ pressed a control button or key 15 ^ nG . 4> ZdTcn a determination b ^TaTto 

to designate this particular tune mark (which was for- „*,^u* „ .u • tu 

merryT^l") Re active time mark 27. When this SUlfJ ^ ^ * 

designation was performed, a vertical reference line 29 SL^JSfS^! "* m "T** ^'.^ dcte V 

paralkl to the layers axis 25 appeared at the location of Z^Z&Tff^ /l^ 1 *^ * 

the active time mark 27. TtTopenitor then "dragged" 20 ^ g ^ m f hm * m madc < d f reCt,y ' Wlthom n P dalm « 

the active timeline 21-A and thfvertical referencTline ^rT^^ ^ "^Z^ , 

29 to its present location using the mouse or other cur- iU ^ ( ? erato 5 B controIlm « ^ machme, lll-Yes, 

sor control device while holding the control button Reared position cursor is updated mandadrtermi- 

d 0WIL nation is made 113 as to whether or not the desired 

Since the reference line 29 now intersects timeline 25 P°^ oa cursor » m one of ^ variable rate areas (for- 
21-B at a point which is closer to the time mark 31 on wa ™ motton area or motion area )- ff **« desi «^ 

the left than it is to the one on the right or either end of position cursor is in one of the variable rate areas, the 
the timeline 21-B, when the operator releases the con- dcared forward rate of motion or reverse rate of motion 
trol button on the mouse (70 in FIGS. 2 and 4), timeline 15 detected (86 in FIG. 5) and the detected rate of mo- 
21-A moves to the left along the time axis 24 to automat- 30 aon 18 converted to a command (56 in FIG. 5) to be sent 
really and exactly align the active time mark 27 on time- 115 to the VTR or other mass storage device (50 in 
line 21-A and the nearest time mark 31 on timeline 21-B. 2). If the cursor was not in the variable rate area, 

Note that for the purpose of time alignment, both ends 113-No, the desired forward rate of motion or reverse 
of the timelines 21 are also treated as if they were time TaUt of motion is detected (86 in FIG. 5) and the de- 
marks 31. 35 tected rate of motion is converted to a command (56 in 

In addition to aligning the timelines 21 and their time PIG / ^ to ^ sent 115 to the VTR or other mass storage 
marks 27 and 31 on the screen, the software also makes device (50 in FIG. 2). 

note of the timecodes and offsets that wil] be required to After the command is sent 115 to the mass storage 
access the under-lying video sources for recording or device, if path Ill-Yes has been followed, or after it has 
previewing in a time-aligned manner. The various 40 * >een determined that the operator is not controlling the 
means used by video editing systems for physically tnanhinr, Ill-No, a determination is made 116 as to 
time-aligning video sequences, once the proper time- whether or not the operator has marked a selection to 
codes and offsets are known, is well known to those create a time mark, as described in connection with 
skilled in the video editing art FIG. 4. If the operator has indicated a desire to create a 

FIG. 8 shows how video pictorial timelines 16 can be 45 tmle mark, 116- Yes, the mark is created 117 by the 
used in conjunction with the simple titn^iitwc 21 in the graphic interface generator 57 (FIG. 4) and placed in 
same Workspace 15 area, such as the layers area 32. storage in the local memory 60 as described in connec- 
This allows some of the video sequences to be seen with tion with FIG. 4 above. 

the extra visual cues provided by the video pictorial Whether or not the operator has marked a selection, 
timelines 16, while other video sequences are shown by 50 116- Yes or 116-No, as shown in FIG. 9C, a determina- 
the simple timelines 21 that save space and allow room tion is made 118 as to whether or not the operator has 
for more layers of video to be displayed. FIG. 8 also identified a time mark as the active time mark (27 in 
shows several inactive current position cursors 73. FIG. 7A). If not, the graphic timeline positions are 
These are left behind at the last position viewed, when converted 126 (FIG. 9D) to timecode values. If the 
the user's cursor 12 is detached from the current posi- 55 operator has designated a new active time mark, 118- 
tion cursor 13 by double clicking of the control button Yes, a further determination 119 is made as to whether 
on the mouse 70. this is the first time through this loop since the operator 

FIGS. 9A through 9D are a simplified logic flow identified the point If this is the first pass through this 
diagram of a portion of the operator interface of the loop, 119- Yes, the closest mark on the timpHnp is 
video editing system of the present invention. Referring 60 identified 120, the active time mark is high-lighted 121 
first to FIG. 9 A, at start, up the screen, the graphics and and a vertical reference line is created 121 through that 
the video window are initialized 100. The main loop of mark. Following these 119-Yes activities, or without 
the program then waits 101 for a frame sync pulse. performing these tasks if this was not the first time 
When the frame sync pulse is received, the graphics are through this loop, 119-No, operator input from the 
refreshed and a new frame of video is written 102 in the 65 mouse is detected and responded to 122. This input, if it 
video window. The status of the machine (video source) is present, is the desired horizontal motion of the time- 
is then polled and the current position timecode is line and the marks and vertical reference fine associated 
stored 103. Keyboard and menu commands are then with it The response 122 is to move the timeline, marks 
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and vertical reference line according to the operator 
input 

Next, it is determined whether or not die switch on 
the mouse is still active 123. If the mouse button is still 
depressed, 123- Yes, the program advances to decision 5 
block 127 .shown in FIG. 9D. If the monsc button has 
been released, 123-No, a determination is made 124 as to 
which mark on another timeline is closest to the vertical 
reference line associa ted with the active mark on this 
timeline. The selected timeline and its marks and verb- 10 
cal reference line are then moved 125 to be aligned with 
the closest mark on the other thwitn^ as shown in the 
transition from FIG. 7A to FIG. 7B. This new position 
information for the selected timeline is then converted 
126 to source timccode values. IS 

The 'user interface is then polled to determine 127 if 
the operator has selected either "record" or "preview" 
actions. If the operator has selected neither, 127-No, the 
program returns to the top of the main loop (FIG. 9A) 
and waits far the next frame sync pulse 101. If either 20 
"record" or "preview" were selected, 127- Yes, then 
commands are sent 128 to play all of the sources that 
were active in the Workspace 15 at the times corre- 
sponding to their locations in the Workspace, thus cre- 
ating the desired combined video sequence. If the se- 25 
lected operation is "preview" 129, the video and audio 
are displayed only 130. If the selected operation is "re- 
cord" 131, the video and audio are recorded 132 to the 
master tape and the corresponding data is added to the 
edit decision list (EDL). In either event, the program 30 
returns to the top of the main loop and waits for the next 
frame sync pulse 10L 

While a preferred embodiment of the present inven- 
tion has been shown and described, it wfll be apparent 
to those skilled in the art that many changes and modifi- 35 
cations may be made without departing from the inven- 
tion in its broader aspects. For example, although the 
operator interface for a video editing system of the 
present invention, has been described as a single inter- 
face with several variations, portions of it could be 40 
modified for other uses and applied to those uses sepa- 
rately. The claims that follow are therefore intended to 
cover all such changes and modifications as fall within 
the true spirit and scope of the invention. 

1 claim: 45 
1. A method for representing a video sequence, hav- 
ing a plurality of video frames, visually and spatially 
while minimizing local memory requirements, the 
method comprising the steps of: 

sampling each video frame of the video sequence to 50 
produce a frame sample for each video frame; 

storing the frame samples for the video frames in a 
local memory; and 

displaying the frame samples as a pictorial timeline 
representing an edge view of the video sequence. 55 

2 A method according to claim 1, wherein frame 
samples ere displayed side by side and the sampling step 



,197 

16 

comprises the step of producing for each video frame a 
plurality of pixels that each represent an average video 
value within a corresponding region along a vertical 
axis of the video frame. 

3. A method according to claim 2, wherein the pro- 
ducing step comprises the step of emphasizing a dra- 
matic video feature within each region. 

4. A method according to claim 1; wherein the sam- 
pling step comprises the step of n^W^ing a plurality of 
pixels along a vertical axis of each video frame. 

5. A method according to claim 4, wherein the verti- 
cal axis is taken through a center of each video frame. 

6. A method according to claim 1 farther comprising 
the steps of: 

storing a corresponding video sequence in a mass 
storage medium; 

providing with each frame sample a pointer to the 
• video frame from which the frame sample is pro- 
duced; 

selecting a location within the video sequence by 
'indicating a frame sample from the pictorial time- 
line; and 

using the pointer from the indicated frame sample to 
access the video sequence at the selected location. 

7. A method according to claim 6 further comprising 
the steps of: 

responding to operator instructions by rearranging an 
order in which the frame samples are displayed; 
and 

outputring an edited video sequence according to the 
• operator rearranged order of frame samples using 
the pointers to the video frames. 

8. A method according to claim 6 further comprising 
the step of providing a view into tile stored video se- 
quence. 

9. A method according to claim 6 further comprising 
the Steps of: 

sampling an audio signal to produce a plurality of 
audio samples which represent an envelope of the 
audio signal; 

storing the audio samples in the local memory; and 
displaying the audio samples together with the picto- 
rial timeline as a visual representation of the enve- 
lope over time. 

10. A method according to claim 9 further compris- 
ing the steps of: 

storing the audio signal in a mass storage medium; 

providing with each audio sample a pointer to a cor- 
responding portion of the audio signal in the mass 
storage medium; 

responding to operator instructions by rearranging an 
order in which the audio samples are displayed; 
and 

outputting an edited audio sequence according to the 

operator rearranged order of audio samples using 

the pointers to the audio signals. 

* * * • « 
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